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ABSTRACT 

A 6~year study of outdoor^based management training 
(OMT) programs used traditional evaluation methods and new methods 
designed specifically for these unique programs. A survey of 1,000 
training directors indicated that this type of training was very 
controversial. The next step was to use traditional training and 
organizational behavior evaluation methodologies to explore the 
impact of a l“day ropes course on group and individual behaviors. 
Evaluation work was based on Kirkpatrick's (199A) four levels of 
evaluation: reaction, learning, behavior, and results. Two types of 
behavior were evaluated through a Likert“type questionnaire: 
individual and group. Results indicated a significant improvement in 
the overall functioning of the work group but no significant changes 
in individual behavior. Some participant variables found to influence 
the ef f ec t iveness of outdoor programs were as follows: 
i n t ac t /nonint ac t work groups, volunteers /nonvolunteers , gender 
composition of groups, supervisor attendance at the program, 
indoor/outdoor, and the facilitators. The eva 1 ua t ion was expanded t o 
use a control group, open-ended questions, focus group interviews, a 
structured on-the-job observation, and participant journals. 
Evaluations of programs in other countries produced evidence that the 
programs had an impact on the overall culture of the organization. 

Two issues for the future were identified: the need to certify 
facilitators and the need to look at the underlying conceptual 
framework. (Contains 10 references.) (YLB) 
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An increasing number of organizations are using outdoor-based management training (OMT) programs 
for individual managers, management teams and work groups. Training organizations throughout the 
world currently offer some type of outdoor training. Participants in OMT range from Fortune 100 
executives to nurses to civic group volunteers. A number of organizations have even developed their own 
outdoor training programs and Universities are sending people "to the woods" as part of their traditional 
education programs. 

Of course, fads in training are nothing new. At one time or another, trainers have tried virtually 
everything in pursuit of more effective managers and teams. But what is so intriguing about the OMT 
movement is the intensity of the debate regarding its utility as a training strategy. On the one hand, 
anecdotal support is voluminous. One need not look far to find former participants who will attest that 
their experience in an OMT program was highly beneficial (e.g., Long, 1987; Broderick, 1989). Several 
published accounts even recant participant statements which suggest that OMT experiences are far superior 
to training in more traditional environments. Even top executives are among the converts. For example, 
Nike Corporation Vice-President Nelson Farris has said, ... "I think every one of our employees should go 
through it, not just some people. We are looking for ways to get people to open their minds and deal with 
the process of change —this program will help our company" (MacNeil-Lehrer, 1989). OMT has also 
evoked fervent opposition. Skeptics contend that such programs are, at best, a waste of time and, at worst, 
may even detract from managerial effectiveness. Ron Zemke, Senior Editor of Training Magazine, 
suggested that ..."outdoor programming is nothing more than an opportunity for organizations to pack 
whole management teams off to risk life and limb together" (1989). Jack Falvey, writing in the Wall Street 
Journal, argued that ..."building outdoor party games and simulations, when the real work to be done is 
ail around, should be grounds for managerial malpractice indictments" (1988). 

Despite the dramatic growth in corporate expenditures on outdoor-based training and the controversy, 
we find that many management and training professionals have little understanding of what outdoor-based 
training is, where it came from, or what it purports to do. At the same time, we have observed that there 
seems to be a pervasive desire to learn more about this intriguing training phenomenon. Most managers 
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we talk to have heard or seen something about "this outdoor training stuff but are reserving judgment 
until they can learn more about what really is involved. Unfortunately, the speed with which outdoor 
training has permeated organizational training and development has surpassed the associated literature on 
the topic. For example, the first standard reference for the manager who wishes to learn more about OMT, 
particularly in the corporate setting was just recently published (Roland, Wagner & Weigand, 1995). 

In an article entitled "Outdoor Training: Revolution or Fad?" (Training & Development .loumal. 
1991) Wagner, Baldwin & Roland point out the need to develop effective research programs to validate 
OMT programs. If these programs are to become long-term, viable training methods, it is essential to 
validate their use. This article summarizes our six year efforts to research OMT programs using 
traditional training evaluation methods, and new methods designed specifically for these unique programs. 
The history of evaluating OMT 

Our efforts to evaluate OMT programs began in the Spring of 1989. Chris Roland of 
Roland/Diamond, Keene, New Hampshire had designed and built a 1-day ropes course training program 
at a U.S. Navy installation in Indiana. After the first year of training Chris contact Tim Baldwin and Dick 
Wagner (then at Indiana University) and worked with them to develop an evaluation of this program 
(training over 300 people per year). We first developed a "benchmarking" study to see how other OMT 
programs were evaluated. An exhaustive literature review in the U.S. and then around the world yielded 
very disappointing results. Only a handful of studies were found, and almost all of these studies were 
flawed in some way. 

Many of the studies were done by the consulting firm which was conducting the program - causing 
a concern about the objectivity of the study. Almost all of the studies were based on a very small sample 
size - sometimes conclusions were based on studies of as few as three trainees. In addition to the small 
sample sizes, many of the studies were based solely on observational data from one observer. Thus, for 
example, many studies consisted of the trainer observing three or four trainees during an outdoor program, 
and writing up his/her comments the program. Research of the this type makes it impossible to validate 
a program as complex as outdoor management training. 
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Initial evaluation efforts 



We had two distinct advantages when we began our evaluation efforts: First, we had a sample size 
of at least 300 the first year, and potentially as big as 5,000; and second, we were associated with a 
University and were not actually conducting the outdoor training programs. In addition, the members of 
the initial research team (Wagner & Baldwin) came from '^traditional" training backgrounds, and were 
not particularly "sold" on outdoor training. This tender to increase our objectivity. 

We began with a survey of 1,000 U.S. Training Directors to try to get a "feel" for what outdoor 
training was all about This survey told us that this type of training is very controversial, and was 
summarized in Training & Development Journal (Wagner, Baldwin & Roland, 1991). Our next step was 
to use traditional training and organizational behavior evaluation methodologies to explore the impact of 
this 1-day ropes course on both group and individual behaviors. 

We have based our evaluation work on Kirkpatrick^s four levels of evaluation (Kirkpatrick, 1994), 
as follows: 



REACTION: How the participants feel about the program. It is really a measure of "customer 

satisfaction". 

LEARNING: To what extent did participants change their attitudes, improve their knowledge and 

increase their skills? 

BEHAVIOR: To what extent did the participants change their on-the-job behavior? 

RESULTS: What final results occurred? This would include such factors as increased sales, improved 

quantity of production, improved quality, reduced costs, reduction of accidents, reduction 
in turnover, increased profits and return on investments. 

The initial goal of our evaluation efforts was to determine what behavioral changes, if any, occurred 
after people had participated in an OMT program, and how people reacted to these programs. Based on 
previous research in the area of team building, we evaluated two types of behaviors: individual behaviors 
(self-esteem, locus of control, faith and confidence in peers); and group behaviors (cohesiveness, clarity, 
homogeneity, problem solving and the overall process of the group). We developed a questionnaire to 
allow us to evaluate these reactions and behaviors on a pre-post-post basis, using either a 5-point or 7-point 
Likert scale (1 = strongly agree, 7 = strongly disagree).. 
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Initial evaluation findings 



The results of these initial evaluation efforts were published in Training & Development Journal 
(Wagner & Roland, 1992). This research has consistently shown a significant improvement in the overall 
functioning of their work group after the group had attended an OMT program. On the other hand, no 
significant changes have been reported in the individual behavioral variables. 

While these evaluation efforts strongly suggested that outdoor management training programs are 
effective in improving group process and interaction skills, we found that this improvement did not occur 
for all participants uniformly. Some of the participant variables that we have found to impact the 
effectiveness of outdoor programs are discussed below. 

Intact/Non-intact work groups: Many indoor management training programs isave involved 
training workers who were strangers to each other prior to the actual training program. The rationale for 
these programs was that strangers often react more honestly with each other than will those who work 
together on a daily basis. Our evaluations show that intact work teams (those who interact at work on a 
regular basis) benefit significantly more by attending OMT programs than do those people from non-intact 
work groups. 

Volunteers/non-voluntecrs; Most training manuals suggest that only volunteers should attend 
training programs. Almost half of the participants in this program did not volunteer to attend the 
program. Despite having been ’’forced" to attend OMT, the behavioral changes in the non-volunteers was 
not signiilcantly different than it was for those who volunteered to attend the program. 

Gender composition of the group: Another interesting finding concerned the gender composition 
of the work groups attending the program. The groups in the programs we studied ran the gamut from 
all female to all male. Most groups contained a varying number of both males and females. We found that 
groups with a balance of both males and females showed a greater improvements than did those groups 
which were either male dominated or female dominated. 

Supervisor attendance at the program; Slightly over one-half of the groups we evaluated attended 
the program with their supervisor. We expected the presence or absence of the supervisor to impact the 

5 




6 



effectiveness of the program. We were surprised to discover that the only significant impact the 
supervisor's presence had on the group was that those groups who attended with their supervisor liked the 
program better. As far as changes in group behaviors, we found no significant difference attributable to 
the presence of the group^s supervisor. 

Indoor versus outdoor; Many people have argued that it is the outdoor setting itself that enhances 
the success of this type of training. Our research does not support this, since the success of our programs 
was unrelated to how much of the training was held outdoors. Our continuing research strongly suggests 
that it is the process, and not the setting which facilitates the behavior changes (Clements, Wagner & 
Roland, in press). 

The Facilitators; A key element in any training effort involves the trainers or facilitators. We have 
consistently found that significant di^erences between program effectiveness can be explained, at least in 
part, by the skill of the facilitator, in two key group behaviors: effectiveness and awareness. For the first 
year of the Navy program facilitator training consisted mainly of on-the-job (OJT) type of training, 
emphasizing primarily the "activity process" and safety aspects of facilitating the outdoor program. Before 
the start of the second yearns program each facilitator participated in an intensive facilitator training 
program, emphasizing two areas in addition to the OMT activities: relationship to on the job activities; and 
debriefing/ human behavior skills. 

A comparison of the two programs indicated that the second year’s program was significantly more 
effective, especially in the area of group effectiveness. Since the only key change from the first year’s 
program to the second year’s program was the training of the facilitators, we felt that the training of 
facilitators in the areas of business and human behavior significantly increased the effectiveness of the OMT 
program. Our continuing research has strengthened this finding that effective facilitation skills for outdoor 
programs includes actual business skills and human behavioi/group process skills, in addition to the 
activities skills. 

Expanded evaluation methodologies 

While the self-report questionnaires provided us a lot of useful and informative data, we quickly 
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became aware of the lack of depth that using a pre>post-post methodology offered us. Our first addition 
was the use of a control group, in addition to the training group. The use of a control group is important 
if we are to make inferences based on any changes found in trained group. The measure of interest is the 
"change score" - the post score minus the pre score. If the trained group had a change of from the pre 
to the post we might be tempted to say that this suggests that the training was effective. However, this 
could be an erroneous conclusion since some "external condition", such as a new manager, a change in 
policy or procedure, could have caused a change for the entire company - with or without training. 
However, if a control group is used, and trained group shows an increase, but the untrained (control group) 
does not, we have a much better case for saying the training program was effective. 

Since, when working with corporate work teams, random assignment to the two groups 
(trained/control) is usually impossible, the alternative solution we often use is the concept of a "stratified 
group". Using this technique, the members of the control group are "matched" to the members of the 
trained team on the basis of potentially important differences. For example, if the trained group contains 
6 females and 12 males, the control group should consist of "approximately" the same mix males and 
females. 

A second method we began to use was to include open-ended, behaviorally oriented questions on 
the post questionnaires. These allowed the trainees to amplify their ratings on the questionnaires, but also 
require them to link this response to an actual job behavior. For example, after giving an item a high 
rating (I feel free to ask my peers for help when I need it), we would ask a participant to tell us why he/she 
gave this rating, and then ask them to give an example from work to justify this response. 

The use of open-ended questions was developed for use in long-distance evaluations. Initially, we 
limited our evaluations to nearly organizations, but as our database grew and we began to publish some 
of our data we also began to evaluate programs all over the U.S. and then internationally. For the local 
programs we next began conducting interviews with the program participants. In some cases we used 
individual interviews, based on questions similar to the open-ended questions discussed 
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above. We also began to use focus-group interviews when time was limited, or we were dealing with large 
groups of participants. 

Along with participant interviews we also used a structured on the job observation to evaluate 
behaviors, and developed a series of behavioral questionnaires given to subordinates, supervisors and peers 
who had not attended the training program. In this way we were able to confirm the rating provided by 
the participants from independent observers who are familiar with the participants behaviors at work. For 
many programs we have successfully used video-taping as an evaluation method. This allows us to review 
the program at some leisure, and to gather comments from a number of trained observers. 

For multiple day programs we have also used participant journals with some limited success. 
Journals filled out several times a day during a 3 - 5 day (or longer) program offers us a great deal of 
information. Two problems we have run into with journals are getting them to be filled out on a regular 
basis during the program, and being able to deal with many pages of data filled out by 15 or 20 
participants in a program. We have found that scheduling a series of specific times during each day to 
complete the journal entries helps to get the journals fiiied out in a timely manner. Dealing with all the 
data can be made easier by structuring a number of specific questions into the journals. This eases the 
problem of interpreting the large volume of data. 

International outdoor management training evaluation 

By 1992 our evaluation efforts in the U.S. had involved over 30 organizations, over 300 groups and 
over 5,000 program participants. In the U.S. our efforts had focused primarily on the impact of outdoor 
management training programs on team-building behaviors. We began finding interest in evaluating 
programs from around the world, including: Canada, South Africa, Australia, New Zealand, Germany, 
Belgium, England and Singapore. 

Evaluations of programs in some of these countries also focused on team-building as a goal. 
However, we also began to find evidence that something beyond team-building was taking place. In 
evaluations in the U.S. and with John Campbell of Executive & Staff Training (EAST) of Harrogate, 
England, we found growing evidence that these programs could have, in addition to their team-building 
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impact, also have a powerful impact on the overall culture of the organization. This includes such areas 
as: style of management (authoritarian versus participative); the empowerment of the employees to act in 
their own best interest at work; and the changing work environment many organizations face as the 
globalization of the work force takes place throughout the world. 

Issues for the future 

After six years of evaluating outdoor management training programs, where do we go from here. 
First of all, we believe that the international efforts will continue to grow in the coming years. 

We also believe the need for new methods of evaluation will be needed. For example, while behavioral 
observations have been the focus of our efforts, we are more and more focusing efforts on Kirkpatrick’s 
fourth evaluation level - results. As more organizations train larger segments of their workforce we are 
better able to evaluate outdoor management training programs using bottom line results like productivity, 
turnover, sales and profits. 

Within the framework of outdoor management training ive see the focus on two issues: the need 
to evaluate, train and certify frjcilitators; and the need to look at the conceptual framework underlying 
outdoor management programs. As discussed earlier, the facilitator can make all the difference in 
determining the effectiveness of the program. Many facilitators of outdoor management programs have 
initially come from an "outdoor" background. They are often hikers, campers, rock-climbers, etc., and 
have adapted these skills to training managers. Unfortunately, many of these facilitators lack in-depth 
organizational or group process knowledge, and their programs have suffered accordingly. Research is 
needed to develop better training and certification programs for facilitators. 

An often related problem is the lack of a conceptual framework for many programs. While we 
commonly hear references to various learning theories David Kolb’? Experiential Learning Cycle is the one 
we hear referred to most often. While many trainers make this linkage, we find that many outdoor 
management programs focus on the "activities" to the exclusion of other critical parts of this theory. 
Future research needs to focus on the conceptual framework of outdoor management training. Only in this 
way can these programs move past the status of a fad and become a training revolution. 
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